Users often interact with large language models through black-box inference APIs, both for closed- and open-weight models (e.g., Llama models are popularly accessed via Amazon Bedrock and Azure AI Studio). In order to cut costs or add functionality, API providers may quantize, watermark, or finetune the underlying model, changing the output distribution -- often without notifying users. We formalize detecting such distortions as Model Equality Testing, a two-sample testing problem, where the user collects samples from the API and a reference distribution and conducts a statistical test to see if the two distributions are the same. We find that tests based on the Maximum Mean Discrepancy between distributions are powerful for this task: a test built on a simple string kernel achieves a median of 77.4% power against a range of distortions, using an average of just 10 samples per prompt. We then apply this test to commercial inference APIs for four Llama models, finding that 11 out of 31 endpoints serve different distributions than reference weights released by Meta.

Snuffling through the underbrush, the shaggy little creature wanders through the sylvan night, sticking its nose in one place, then another, seeking the aroma of its soft-bodied dinner. The forest is dark and the pixie's eyesight poor, but long whiskers and a keen sense of smell allow it to get around. Threatened, it takes off at breakneck speed, barreling through the vegetation, ducking through holes, soon lost from sight. Many animals spend their nights cruising the forest floor, searching for small prey in a similar fashion: Hedgehogs, shrews, weasels, to name a few, and bigger ones, too, like opossums and even pigs. The world is full of them. But this one is different. All the others are hairy. This one's pelage is also soft, made up of millions of thin strands. All the others move about on four legs and bear live young. And as the male calls, he identifies himself: "Kee-wee, kee-wee."